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1 . 


Introduction 


Scotland's colleges provide education and training for around 400,000 
learners every year. For some learners, it will include programmes of full-time 
study and nationally recognised awards. Most learners will be involved in 
part-time study and many will also be in employment. Some will be second or 
third chance learners whose initial steps may be small and it may be difficult 
to measure the extent of their progress. 

Success on a programme can be viewed in different ways. The hard 
outcomes of learner retention and attainment give important measures of 
success in terms of the number of learners gaining programme awards. But 
this only gives a partial view. These learners will also have important 
achievements in terms of gaining wider skills and making progress from 
previous learning. For many other learners, their real success is in this latter 
kind of achievement and progress, even if they fail to complete the whole 
programme - and for these learners sometimes the educational journey will 
take a longer period of time. 

The annual collection and publication by the Scottish Further and Higher 
Education Funding Council (SFC) of college performance indicators has 
focused attention on what these indicators actually tell and whether they 
provide a fully comprehensive and meaningful story of what college learners 
achieve. Since 2005 HMIE has overtly made more holistic evaluations of 
learner progress and outcomes^ and graded accordingly during external 
reviews. At the same time, colleges have been working to strike a balance 
across the various aspects of learner progress and outcomes of learners in a 
real attempt to form an overall measure of learner success. 

HMIE has consulted widely and engaged with Scotland’s colleges to reach an 
understanding of the value and use of performance indicators (Pis) and other 
measures to assess learner progress and outcomes and to capture the views 
of colleges. This report summarises and comments upon the practices of 
Scotland’s colleges in the use of performance indicators and their 
developments in recording and assessing the wider achievement of learners. 
The report also makes recommendations on the use of Pis within colleges 
and on the development of national measures of learner achievement. 


^ Learner progress and outcomes is element A7 of the SFC/HMIE Quality Framework for 
Scotland’s Colleges, May 2004 
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2. Methodology 


The report draws on a range of evidence, including from 43 HMIE college 
reviews carried out since January 2005 and other HMIE contacts with 
colleges. Desk research on reports and literature from the college sector in 
Scotland and the rest of the UK formed essential background evidence for the 
report. 

A substantial questionnaire provided much evidence for this report. It was 
completed and returned by 33 of Scotland’s 43 colleges (77%). Visits to 
colleges helped to confirm initial findings and evaluate good practice for wider 
dissemination. 

Members of the Scottish Further Education Unit (SFEU) Quality Community of 
Practice (formerly the Quality Improvement Forum) provided useful 
contributions to a range of issues identified by the task. Staff from SFC’s 
statistics branch also provided valuable information to inform this task. 

Finally, a dissemination event to share initial findings of the report attracted 
contributors from 23 colleges. Workshop activities helped to shape aspects of 
its content. 

A full list of those colleges that provided views and information to inform this 
task is given in Appendix 5. 
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3. Summary of findings 


Scotland's colleges have used a range of Pis to measure learner retention 
and attainment on programmes of study since the early 1990s. Almost all 
colleges used PI data to support programme self-evaluation to inform actions 
for improvement in portfolio planning. More than a few colleges also reviewed 
and evaluated overall attainment at college level using PI data. 

Since 2001 , SFC has routinely published an annual set of college sector Pis. 
Colleges questioned the usefulness of these aggregated Pis for individual 
institutions and the meaningfulness of cross-college comparisons at this level. 

Colleges had mixed views on the programme attainment Pis. They 
acknowledged the value of the student outcomes PI, particularly for learners 
partially achieving a group award. However, staff familiarity with the SPAR PI 
and its historical use in trend analysis in more than a few colleges were 
barriers to moving over to the student outcome measure. 

Colleges had started to use a range of approaches to capture the progress 
and wider achievements of learners as a result of a programme of study. 
Almost all colleges used some form of Individual Learning Plan (ILP) as the 
tool to record and assess learner progress. Most colleges also used ILPs to 
stimulate progress and achievement and involved learners in self-assessment 
and goal-setting. 

Colleges had started to use a mix of objective measures, semi-objective 
measures and subjective assessment to evaluate the overall distance 
travelled by learners. Most colleges thought that the development of national 
measures of distance travelled was desirable, although a number of 
significant barriers to development were acknowledged. 

About half of colleges were involved in benchmarking in partnership with other 
colleges and most colleges also found the SFC benchmarking tool helpful. 
Almost all colleges had embedded some form of trend analysis covering 
aspects of attainment, progress and achievement into their systems and 
processes, and this analysis had the capacity to inform teams at all levels. 
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4. Learner progress and outcomes in context 


The 1990 HMIE report Measuring Up: Performance Indicators in Further 
Education proposed that when Pis are used as evaluation tools, they can 
serve one or both of two broad purposes. When used formatively, they can 
assist college staff, at all levels, to improve the quality of provision. When 
used summatively, they can provide a basis on which an opinion of the 
overall health of an institution’s education provision may be formed. More 
specifically the report proposed that Pis could be used in combination to: 

• inform the planning process and put resource allocation on a more 
realistic footing; 

• provide a focus for the involvement of academic planning; 

• assist ongoing monitoring of effectiveness and efficiency; and 

• evaluate educational provision and its quality. 

In 1993, the follow-up HMIE publication On Target: Using Performance 
Indicators in Colleges of Further Education developed the use of Pis and their 
role as a tool for analysis and decision making at departmental/section and 
college levels. The report concluded: 

Student experience is at the heart of any attempts to improve the 
quality of vocational education and training. The quality of that 
experience needs to be carefully monitored and care should be taken 
to evaluate it using the whole range of Pis and not emphasising a 
single indicator at the expense of others. 

Since these reports were published the college sector has undergone 
considerable changes, including incorporation and the formation of the 
Scottish Further and Higher Education Funding Council. 

In Improving Scottish Education (2006) HMIE reported that: 

Colleges have been successful in encouraging lifelong learning through 
a wide range of flexible arrangements which have led to learners 
achieving well and developing core skills, personal skills, vocational 
skills and other skills for employability. 

However, despite developments in quality improvement arrangements 
over the last few years, weaknesses are still prevalent in the 
approaches of some programme teams and colleges to quality 
assurance and improvement activities. Programme teams often make 
insufficient use of programme and unit attainment data in the analysis 
and evaluation of the effectiveness of programme delivery. 

SFC (and formerly the Scottish Further Education Funding Council (SFEFC)) 
has produced an annual set of college sector Pis since 2000-01 . This 
practice was introduced in response to the then Scottish Executive stating 
that, “good, robust and relevant performance indicators at college level are 
essential measuring tools for the promotion and sharing of good practice 
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amongst colleges”, and that the Council should, “review the range of Pis 
considered necessary, and then put in place appropriate mechanisms to 
establish, measure and publish these.” 

The current set of Pis deals with measures such as student induction, 
retention, student outcomes and achievements, staff qualifications and 
financial health. The most recent annual publication Student and staff 
performance indicators for further education coiieges in Scotiand 2006/07 
(August, 2008) restates the purposes of these Pis, which: 

• provide better and more reiiabie information on the achievements and 
impact of Scotiand’s coiieges; 

• inform coiiege seif-evaiuation to iead to seif-deveiopment; 

• enabie individuai coiieges to benchmark their own achievements 
against that of other coiieges and thus iead to quaiity improvement; 

• inform HMIE reviews; 

• inform poiicy deveiopments; and 

• enhance accountabiiity for use of pubiic funds in coiieges. 


Use of Pis for quality improvement 

Colleges use Pis as a helpful aid and focus for self-evaluation by programme 
teams and cross-college teams responsible for areas such as guidance and 
support. Used correctly, the data stimulate staff to ask appropriate questions 
to identify strengths and weaknesses at programme and college level and 
then plan actions to address identified areas for development. However, it is 
important to be able to interpret PI data and contextualise these indicators as 
guides or signals of the quality of their provision, rather than absolute 
measures. 

HMIE’s Effective Seif-Evaiuation Reporting in Scotiand’s Coiieges (October, 
2007) stated: 

Aimost aii coiieges have highiy effective systems for providing accurate 
up-to-date Pis on iearner retention and attainment for teaching teams 
to anaiyse in seif-evaiuation reports. Most seif-evaiuation reports by 
teaching teams at ieast acknowiedge iow vaiue Pis. Where reports 
anaiyse underiying reasons for these in more detaii, these sections 
usuaiiy demonstrate detaiied staff knowiedge of individuai iearners’ 
circumstances. 

Routinely, colleges use a suite of Pis to inform self-evaluation by reference to 
measures of: 

• learner retention: Student Retention Rate 1 (SRR1), early retention. 
Student Retention Rate 2 (SRR2) and student retention; 

• programme attainment: Student Programme Attainment Ratio 
(SPAR), and student outcome; 

• unit attainment: Student Achievement by Unit (SARU); and 

• post-programme success: Post Course Success Ratio (PCSR). 
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Definitions of these Pis are given in appendix 1 and chapter 5 of this report 
discusses their use in colleges. A useful description of the main PI measures 
is given in the SFC publication Student and staff performance indicators for 
further education coiieges in Scotiand 2006/07 (2008). 


Benchmarking tool 

The Scottish Funding Council has developed an analytical tool to assist 
college staff with responsibilities for quality assurance and enhancement and 
teaching staff in their analysis of the statistical data the Council provides on 
learner retention and attainment. The analytical tool is based on an Excel 
spreadsheet that uses a series of filters to select data in a particular subject 
area or programme and to enable the user to compare data with those from 
other colleges. 


The use of Pis during external review by HMIE 

HMIE review teams use Pis to: 

• inform evaluations of learner retention and attainment; 

• provide evidence of the impact of college policies and strategies; and 

• establish how effectively staff use Pis as tools for improvement, to 
contribute to the teams’ evaluations of outcomes for learners and the 
effectiveness of self-evaluation in colleges. 

Over the period 2004-08 subject reviewers analysed data on learner retention 
and attainment to inform their evaluations using Ql A7.2 Learner attainment. 
They informed their judgements about A8 Guidance and support by reference 
to data on early retention, learner retention and post-course success. 
Cross-college reviewers used PI data to provide evidence of progress the 
college had made against key aims and objectives (Ql B1 .4 Achievement of 
educationai aims, objectives and targets). They used PI data to explore 
college arrangements for monitoring the impact of equality policies including 
analysis of learner applications, admissions, level of achievement and 
reasons for leaving by gender, age, disability and race (Ql B2.2 Arrangements 
for access and inciusion). Reviewers of guidance and support used PI data to 
explore with college staff the reasons for low values for early retention and 
learner retention levels and the effectiveness of strategies to improve 
retention and the impact of interventions made. Reviewers of B6 Quaiity 
assurance and B7 Quaiity improvement exp\ore6 staff use of Pis to provide 
evidence of the effectiveness of self-evaluation for quality improvement in 
colleges. They investigated programme team reports, subject area 
evaluations, analysis of trends, and interventions to address low retention and 
attainment. 
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Capturing an all-round picture of attainment and achievement 

The current measures do not reflect the full picture of the learning, 
achievement and attainment that takes place in colleges. For example, these 
measures cannot : 

• recognise additional achievements and give credit for successes 
outside the conventional measures, possibly against personal learning 
goals; 

• give credit for partial attainment (although the student outcome PI does 
go some way towards addressing this); 

• recognise soft skills; and 

• recognise distance travelled. 

Colleges have attempted to give structure to forming a more holistic view of 
overall progress and attainment by considering not simply attainment of a 
programme award or units, but the broader set of skills a learner may have 
achieved during their period of study. The Scottish Government’s Skills for 
Scotland: a Lifelong Skills Strategy {2007) defines skills by focusing primarily 
on several overlapping clusters of skills; 

• personal and learning skills that enable individuals to become 
effective lifelong learners; 

• literacy and numeracy; 

• the five core skills of communication, numeracy, information 
technology, problem solving and working with others; 

• employability skills that prepare individuals for employment rather 
than a specific occupation; 

• essential skills that include all of the above; and 

• vocational skills that are specific to a particular occupation or sector. 

The strategy document also identifies softer skills, which are less definable, 
but that employers value as vital to the success of their organisations. These 
include: 

• effective time management; 

• planning and organising; 

• effective written and oral communication skills; 

• the ability to solve problems; 

• being able to undertake tasks or make submissions at short notice; 

• the ability to work with others to achieve common goals; 

• the ability to think critically and creatively; 

• the ability to learn and to continue learning; 

• the ability to take responsibility for professional development; and 

• having the skills to manage or be managed by others. 

This more holistic view of overall achievement and attainment has also been 
reflected in the SFC/HMIE Quality Framework (May 2004) and in external 
evaluation of learner progress and outcomes by HMIE. 

The quality indicator for learner progress and achievement (A7.1) 

considered the progress made by the learner from prior attainment. 
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achievement, learning and experience as a result of a programme of study. 
This would include progress towards learning goals, progress from the start of 
the programme and progression towards further study and employment. It 
would also include progress of a learner in gaining skills for learning and 
employment. 

The quality indicator for learner attainment (A7.2) considered outcomes of 
learning such as attainment of formal qualifications, attainment of industry 
related awards and tests, retention, post-course success including 
progression to HE or relevant employment, and success in award schemes 
and competitions. 

The wider achievement and progress across the areas covered by quality 
indicators A7.1 and A7.2 is often referred to as distance travelled. 


Measures of achievement and distance travelled 

Most colleges recognise the value in developing some measure of wider 
learner achievement to complement attainment measures. Butcher and 
Marsden’s article Measuring Soft Outcomes: A Review of the Literature 
(2004) reported: 

The iiterature to date has emphasised the need to measure soft 
outcomes as evidence of a stepping stone Unking non-accredited/soft 
iearning with progression to a hard outcome, such as a job or taking an 
accredited educationai course. Such an approach narrows the 
interpretation of sociai inciusion to empioyabiiity and educationai 
achievement and ignores vast swathes of the community and voiuntary 
sector working with ciients who may never access jobs or educationai 
quaiifications. There is a vaiue to society when informai iearning ieads 
to increased motivation and feeiings of responsibiiity, confidence and 
seif-esteem, interpersonai skiiis, improved individuai appearance, time 
management and teamwork for the individuai. 


For measures to assess wider achievement to be developed and nationally 
accredited a number of issues need to be considered in developing 
monitoring systems, including: 

• understanding what the process of developing a soft outcome 
monitoring system would require and how much effort it would take; 

• deciding on what to monitor and what systems to use; 

• deciding how to assess; 

• establishing baselines of learners’ soft skills; 

• reviewing learner progress to assess distance travelled; 

• using a credit framework (such as SCQF) with no prescribed 
curriculum but with clear routes and opportunities for progression; 

• using a common tool such as an individual learning plan (ILP) as a 
means of agreeing goals and recording progress as well as a tool for 
discussion; and 
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• ensuring a nationally recognised system of certification to record 
SCQF-levelled credit achieved by individuals or to record levels which 
learners are working towards. 

This approach places learners and their aspirations and goals at the centre of 
the learning process. While it is the responsibility of professional educators to 
encourage learners to progress and fulfil potential, it is important to recognise 
that formal awards or employment may not be what individuals require from 
their learning. The absence of such a clear end point does not invalidate the 
progress they have made through learning. Learners decide for themselves 
what their goals are (employment, progression to further learning, or personal 
achievement and development) and of what their learning should consist. 
They should furthermore be involved at every stage of the planning, 
monitoring, assessment and accrediting process. 

An important outcome of assessing distance travelled is that learners are 
enabled to identify personal change in their lives as a result of their learning; 
to articulate their different perceptions and insights into their lives, 
relationships and self-image. As well as being assisted in describing these 
changes, individual learners should be supported in capturing their formal and 
informal, mediated and unmediated learning experiences. 


Assessing achievement 

For a learner, the potential outcomes of a system for assessing wider 
achievement include employability, progression to other learning, more active 
participation in communities and personal change. Individual learning plans 
need to be designed in a way to capture learners’ aims, aspirations and 
outcomes, and learners’ perceptions of themselves, their histories and their 
futures. The plans are likely to include those overlapping skills clusters and 
softer skills identified in the Skills for Scotland strategy document. 

A number of collection methods can be used to capture learners’ aims, 
aspirations and outcomes which focus on the broader context of the learner 
and their perceptions of their situation, including self-completion 
questionnaires, tests, portfolios, reviews and recorded observations. These 
collection methods may involve reviews, observations and peer assessments, 
as well as tutor and self-assessment. Scales of feeling or agreement with 
statements, comparisons with previous benchmark data, self-reflection 
reports, and assessors’ views are all useful in measuring soft outcomes. 


Validating achievement and distance travelled 

All systems of monitoring and recording achievement should be subject to 
local audit and robust quality assurance procedures and external monitoring. 
This should include a process of validation and certification that recognises 
vocational experience and skills, entrepreneurship and initiative taking, 
volunteering, team and community involvement, and other factors. These 
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systems should also represent skills in terms of employability, transferability to 
vocational or academic further study and personal change and achievement, 
or a combination of different elements that can be built upon at a future time. 
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5. Learner attainment 


Measures of attainment 

Colleges used attainment Pis in a variety of ways. Almost all colleges (94%) 
used the early retention measure or SRR1. Most colleges (82%) used the 
student retention measure and a majority of colleges (67%) still used the 
SRR2. Some colleges made use of both the student retention and SRR2 Pis. 

Most colleges (67%) continued to use the SPAR as a measure of programme 
attainment and the majority of colleges (70%) were making use of the student 
outcome. Almost all colleges (91%) used the SARD as a measure of unit 
attainment on programmes. Some colleges were using both the SPAR and 
student outcome as measures of programme attainment. 

College comments on the use of the student outcome measure included: 

• student outcomes have been used rather than SPAR for several years; 

• we have placed less emphasis on student outcomes until recently; 

• student outcomes are considered for the overall picture and for 
benchmarking; 

• student outcomes are considered for overall picture - more useful than 
SPAR and SRR2 for flexible learning; and 

• we ceased using SPAR In course evaluations as SFC did not publish It 
In Its publication of Pis In 01-02 and has not printed It In any of the 
subsequent annual publications. We were surprised when we realised 
that HMIE were still using It as a measure of programme success In the 
current cycle of reviews. 

A majority of colleges (61%) used the PCSR to report on post-course success 
of learners. Most colleges reported difficulty in collecting the data and 
questioned their robustness. However, more than a few colleges had put 
considerable effort into collecting and reporting destination data. One college 
reported a 90% response rate from learners and had developed a progression 
wewerthat incorporated employment, articulation and progression 
information, which could then be used by all staff for programme planning, 
evaluation, marketing and other purposes. 

Some colleges used additional measures to demonstrate learner attainment, 
including targets set by other partners (eg ESP) and grades from external 
examination results. A few colleges reported on learner success in national 
competitions. 


Use of the measures to evaluate learner attainment 

Almost all colleges used attainment Pis at programme team meetings and in 
programme team self-evaluation reviews, followed by collation and scrutiny at 
section/department level, and college-wide scrutiny by senior managers and 
committees such as academic boards. Colleges also used attainment data to: 
• identify concerns; 
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• inform actions for improvements; 

• identify good practice; and 

• inform curriculum portfolio planning. 

Programme teams discussed attainment measures at regular meetings, to 
monitor attainment on an ongoing basis. These measures were also used at 
more formal meetings - usually held twice or three times a year. Finally, 
almost all colleges considered and reported on the measures as part of the 
annual programme review process. 

In more than a few colleges, programme teams undertook some 
benchmarking of their Pis and set targets against which Pis were evaluated. 

In a few colleges, programme teams used Pis to monitor retention and 
attainment trends over time. 

At section/departmental curriculum area level, a majority of colleges 
implemented collated subject area review and reporting, based on data from 
the programme teams. In a few colleges, this process also incorporated 
benchmarking, target setting and monitoring trends. 

More than a few colleges reviewed and evaluated overall attainment at 
college level, including scrutiny by Boards of Management. In a few colleges 
this process also incorporated benchmarking, target setting and analysing 
trends over time. 

More than a few colleges used attainment data to identify concerns and 
inform actions for improvements at programme, curriculum area and college 
levels. A few colleges used low PI values to trigger investigation and remedial 
action, making use of traffic lights or heat maps to inform programme teams 
and managers. 

A few colleges used attainment Pis to assist identification of good practice at 
curriculum area level or college level. 

More than a few colleges made use of attainment data to inform planning, 
generally relating to the college curriculum portfolio or programme design. A 
few colleges inserted the data into operational plans, as a basis for planning. 

A few others used the data to inform decisions to change programmes or 
delivery methods. 


Use of Pis for external comparisons 

Most colleges were not convinced about the usefulness of the publication of 
aggregated Pis for individual institutions and the meaningfulness of 
cross-college comparisons at this level. They acknowledged the need for 
accountability for the use of public funds but questioned the interpretation of 
PI measures and how this interpretation relates to a perception of college 
effectiveness, since an external audience is unaware of factors affecting the 
values of PI measures. For example, learners leaving a programme prior to 
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completion because they had gained employment would impact negatively on 
retention and outcome measures, although the learner may regard the 
outcome as successful. 


Collection and processing data on attainment 

Annually colleges collect and process thousands of pieces of data on learner 
attainment. All colleges involve administrative support to assist with this 
process - usually referred to as management information systems (MIS) or 
student records. 

Colleges made a distinction between formal, signed off, validated data and 
the data that were available to programme teams or departments as a result 
of their day-to-day contact with learners. In more than a few colleges, staff 
were able to access their data in real time, as it was keyed in, and/or by 
interrogation through specific MIS reports. 

A number of colleges recognised that certain data were only of use at certain 
points in the academic year, and made arrangements to have the data issued 
at these most useful times. One college, for example, noted that its guidance 
tutors used real-time information weekly, while programme teams used it 
quarterly. Another provided real-time data on retention only. 

A majority of colleges issued data at the end of each teaching block and/or at 
the beginning of the following academic session. This was the fully validated 
and checked data that was used for SFC returns and publications. 

None of the colleges mentioned learners as potential users of data and how 
learners might be issued with, or might access, data. 


Ensuring accuracy of data 

Nearly half of colleges reported some form of liaison and crosschecking 
between the academic team/department and the MIS department to ensure 
accurate data. More than a few colleges involved their quality 
unit/staff/system in ensuring accuracy, including formal auditing. One college 
noted that an online process for entering learner results had significantly 
reduced error. 


Evaluating attainment in non-nationally recognised qualifications, 
work-based and open/online learning 

More than a few colleges had tried to provide a coherent framework for 
assessing non-nationally recognised qualifications, for example by: 

• developing college/locally devised units; 

• providing college certification; and 

• concordat rating their provision against the SCQF. 
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Where the non-nationally recognised programmes were delivered through 
units, colleges generally used the 70% success criterion to signify successful 
programme attainment. However, a few colleges felt this was a crude 
measure and two colleges set their criteria in ways that they considered would 
more accurately reflect the extent of learner success. These colleges argued 
this to be fairer for more vulnerable learners for whom success might 
incorporate factors much wider than simply passing assessments. 

Several colleges referred to the use of ILPs (and PLSPs in the case of special 
programmes) and soft skills attainment. A few colleges mentioned that 
progression to a higher level of study or employment, or course completion, 
was taken into account in determining attainment on special programmes. A 
few colleges referred to the achievement of individual learning goals. 

Overall, a picture emerged of colleges attempting to provide a clear evaluation 
of success (which was measurable and of value to the learner) through such 
things as unitised programmes and established criteria. Where this was not 
possible or appropriate, colleges used ILPs, targets and goals against which 
to measure attainment and achievement. Attainment rates in such cases 
were normally evaluated through the programme review system in line with 
any other programme. 

Colleges generally assessed and evaluated work-based programmes as part 
of the standard programme review processes. This was predominantly SVQ 
provision, but also covered awards by other bodies and also un-certificated 
work-based learning (WBL). WBL was frequently undertaken through 
certificated units and awards, for example SVQs, which incorporated set 
procedures for evaluation. Milestones and learner outcomes were another 
way of evaluating attainment, and there was some online tracking of progress. 
One college incorporated the recording of core and employability skills in 
evaluating work-based learning. 

Some colleges found difficulties in collecting data for rolling programmes, with 
one college introducing fixed sessional dates. Another college carried out 
additional separate cross-sessional analysis. 


Lack of consistency in data coiiection 

This issue of lack of consistency across colleges in data collection was raised 
by about half the colleges. There were clearly strong feelings, particularly in 
relation to accuracy and unfairness. These colleges had concerns not 
particularly about the types of measures used, but about the variation in the 
ways the data were collected, leading to questions about the accuracy of 
national data and the validity of comparisons and benchmarking. The main 
problem was the point at which learners were entered for units and 
programmes - some colleges entered learners at the outset of the 
programme while others did so much nearer the point of programme 
completion. This resulted in lower Pis for the first group of colleges. 
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A related issue was difficulty in managing the administrative procedures in the 
case of learners choosing options, changing programmes or programme 
content, and withdrawing. This also was seen as leading to inaccuracies and 
inconsistencies. Some colleges also perceived variations in practice in 
recording enrolments, retention and student outcomes in other colleges. 

Overall, the sector lacked confidence in the accuracy of the data it collects, as 
a result of inconsistent practices across the sector. It would be helpful to 
explore how accuracy and consistency of data entry can be improved, to 
increase the sector’s confidence in nationally published statistics and the 
benchmarking associated with it. 


Views on the attainment measures available 

Overall, colleges were positive about the current attainment measures. More 
than a few were overtly positive, using terms such as simple but helpful, 
robust, suitable for purpose, satisfactory, sound basis for quantitative analysis 
and measurement of internal and external trends, and useful within an 
organisation for trend analysis and benchmarking. 

More than a few colleges specifically indicated satisfaction with the 
SRR1 /early retention measures and SPAR measure. Opinion was evenly 
split between the few colleges who expressed a preference for either the 
SRR2 or student retention measure. 

Only one college was negative about all of the measures. They had concerns 
about what this saw as the ‘crude’ attainment measures and the limitations of 
the data set. It took the view that the current data could not be improved and 
that a radical revision was required to develop a range of indicators/measures 
that more accurately reflect the impact of the college experience on the 
learner. 

Colleges had mixed views about the use of the student outcome measure. 
Student outcome gave more comprehensive measures of learner attainment 
without acting as a proxy for learner retention and was viewed by some 
colleges as more meaningful than the SPAR for looking at flexible modes of 
learning. The main barriers to the transition from SPAR to student outcomes 
were: 

• staff familiarity with SPAR; 

• currently trend analysis in colleges is based on SPAR data; and 

• staff-perceived complexity of student outcomes. 

More than a few colleges questioned the value of the PCSR measure. Low 
levels of questionnaire returns and data accuracy were seen as major issues. 
However, a few colleges found the data extremely useful for an internal 
quality-related measure and for programme promotion. 
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Interpretation 

More than a few colleges had concerns about the meaningful interpretation of 
PI data without background contextual information. Some of these issues 
were: 

• raw data do not indicate the profile of learner groups and social 
inclusion increases the risk of low Pis, thus true measures of learning 
and progress are not reflected; 

• low SPAR and SARU values can be misleading if viewed as measures 
rather than as indicators of a successful programme; 

• SARU is useful for detailed programme analysis but less so when 
aggregated; 

• measures are distorted by early leavers, for example those taking up 
late offers of university places or employment; and 

• small increase/decrease in numbers can skew percentages in 
programmes with low numbers and changes are often treated as if they 
were statistically significant when they drop or rise. 

Most colleges used a process of scrutiny and exploration where there are low 
PI values, and the context taken into account in evaluating the results. The 
question of interpretation is more likely to be an issue where data are 
published more widely and no context is provided to surround the PI 
measures. 
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6. Learner progress and achievement 


The majority of colleges (60%) recorded evidence of learner progress in all 
five broad skill areas identified in the SFC/HMIE Quality Framework: 

• personal and learning skills - self-confidence and self-esteem; 
self-awareness and self knowledge; attitudes to learning; self directed 
learning and other learning approaches; analysis and critical thinking; 

• core skills - communication, numeracy, information technology, 
working with others and problem solving; 

• vocational skills - competency and proficiency in the 
subject/vocational area; 

• employability skills - in addition to the above: planning and organising 
skills; customer handling skills; self-management; and creativity; and 

• citizenship skills - developed through participation in programmes 
and wider aspects of college life. 

A few colleges indicated that not all areas were recorded equally 
systematically, and a few used the term soft skills to describe some of the 
areas. 


How the evidence is collected, recorded and used to assess learner 
progress 

Personal and learning skills were generally recorded in a Personal 
Development Plan (PDP), Individual Learning Plan (ILP), or similarly named 
document (the acronym ILP will be used for all these documents in this 
report). One college used SQA guidance units for FE-level learners and PDP 
units for learners on HN programmes. Another college used an online 
self-evaluative resource that learners completed at the start and finish of their 
programme. A few colleges used systems that involved self-assessment of a 
range of personal skills. 

A majority of colleges carried out core skills profiling or initial diagnostic 
assessment on learners. Most colleges generally recorded core skills through 
the usual MIS procedures for unit attainment. Routinely this was restricted to 
communication, numeracy and information technology. 

All colleges also recorded vocational skills through these MIS procedures for 
unit attainment. A few colleges used additional sources of evidence such as 
employer reports, placement logs and reflective diaries to demonstrate learner 
progress. 

Colleges recorded skills for employability and citizenship in various ways. 

More than a few colleges made use of ILPs. Other individual colleges used 
personal, social and development (PSD) units, work placement units, 
evidence from programme tutors and guidance tutors, and one college used a 
soft skills grid. One college had defined key competences for employability 
and used periodic learner self-assessment to record progress. 
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Almost all colleges used ILPs as the vehicle for recording progress and 
achievement, supported by periodic reviews with a guidance tutor to discuss 
progress, plan any support required, and provide feedback to the learner. 
Colleges routinely used Personal Learning and Support Plans (PLSPs) as 
carriers of progress and achievement information for learners on Dominant 
Programme Group 18 provision. 

ILPs use within colleges varied: 

• in the coverage of learners, with some colleges including only FT 
learners; 

• in the nature of the plans, to match the requirements of different groups 
of learners (trainees, FE, HE, introductory, special programmes, etc); 
and 

• as a result of initiatives by the particular curriculum sections or 
programme teams using them. 

Across colleges, perceptions of how the ILPs were to be used varied. Some 
colleges viewed them as a pragmatic record of progress and outcomes. 

Others saw them more as a means of promoting learner understanding and 
ownership of their progress and providing motivation for further development 
of transferable personal and employability skills. In most colleges learners 
were encouraged to use the plans to reflect on their learning and their 
progress made. 

More than a few colleges were developing learner self-assessment 
approaches focused on learner ownership and self-development. A few had 
built on some of the Focus on Learning^ initiatives. 

One college had been involved in an extensive set of developments, including 
mapping all activity to SCQF competences, working with their MIS software 
supplier to explore a distance travelled measure based on individual student 
entry and exit levels based on SCQF equivalences. The same college had 
just completed the first phase of a collaborative project with SQA on the 
development and distanced travelled by learners in gaining core skills at HN 
level through delivering a PDF unit via its virtual learning environment (VLE). 
The project was currently in a second phase. 

Overall, the picture is one of colleges moving towards the effective use of ILP 
both to record and to stimulate progress and achievement in the areas cited 
in the Quality Framework, involving the learner in self-assessment and 
goal-setting. However, the pace of movement, the progress made by 
colleges, and the investment in development work all varied greatly. 

More than a few of colleges also used data on learner progress and 
achievement to inform the programme review process, and subsequently 
contribute to college-wide statistical reports and inform improvements in 
teaching and learning. 


^ Focus on Learning 2: Steps to Employability, an SFC/SFEU action research project. 
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Assessing distance traveiied 


Most colleges determined prior attainment and achievement of learners using 
a range of approaches, including: 

• prior qualifications and certification including the learner’s SQA record; 

• discussion at pre-entry interview to explore prior learning and 
experience, and assess level of ability; 

• core skills testing, diagnostic testing and profiling; and 

• guidance interview and induction process - often used as the focus for 
setting up the ILP, which could include self-assessment. 

In addition, a few colleges used: 

• evidence of performance - art portfolio, drama audition, essay or 
written exercise; and 

• vocational skills assessment. 

Colleges used a range of approaches to assess attainment and achievement 
of learners at the end of a programme. These generally fell into the following 
categories: 

• qualifications attained and PI data; 

• pre-exit guidance process: including review, the guidance units 
attained, changes in soft skills, attainment of goals set in ILP, and 
self-assessment; 

• exit core skills profiling; 

• citizenship and workplace awards, and activities to assess progress in 
citizenship and employability; and 

• progression to employment or further study. 

One college also included views of parents/guardians/carers on the 
achievements of their children undertaking a range of college-based 
programmes. 

These categories were a mix of objective measures (qualifications, core 
skills certification, some forms of progression), subjective assessment (soft 
skills progress, achievement of personal goals) and semi-objective 
measures (citizenship and workplace awards). 

One college had the view that distance travelled was recognised, but not 
assessed, while another acknowledged that it was only possible to assess 
distance travelled if there were clear and quantified start and end points. A 
few colleges referred to quantifiable measures, in terms of assessing distance 
travelled by the attainment of units. 

Most colleges assessed distance travelled in terms of the learner and tutor 
reviewing individual learner achievements, including assessment of progress 
against goals. Only one college mentioned measures of distance travelled in 
respect of its mechanism for measuring progress towards employment. 
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Overall, where colleges made serious attempts to assess distance travelled, it 
was generally done informally and on an individual learner basis, employing 
techniques of self-assessment and reflection. 


Development of national distance travelled measures 

Most colleges thought that the development of national measures was 
desirable, although the majority of these colleges gave some form of 
response with caveats. This included almost all of the colleges that had 
engaged in significant development work on progress, achievement and 
distance travelled, and these colleges had evidently drawn on their 
experiences of trying to measure distance travelled to inform their responses. 

Comments by those colleges who thought it desirable included: 

• the use of CGLI awards in employability; 

• helps benchmarking; 

• reinforces the significance in promoting wider access to education; 

• helps to quantify qualitative information, evidence source for HMIE; and 

• there is a need for nationally recognised measures that more 
accurately reflect the importance of the college experience. 

College that thought it was not desirable, or expressed an overall negative 
view, made the following points: 

• too complicated; 

• would be more meaningful to develop a toolbox to support colleges in 
quantifying distance travelled; 

• national application doubtful; 

• aggregated national comparisons not useful; 

• distance travelled is a measure for individuals; 

• Focus on Learning 2 project findings - no single tool could be 
recommended for general use, such is diversity of learner needs (SFC 
Circular 15/2007); and 

• flaws in existing Pis could be magnified. 

Almost all colleges recognised the potential benefits to learners of a distance 
travelled measure in terms of recording learner confidence and self-esteem, 
self-reflection, motivation, recognition and evidence of employability and soft 
skills. A few colleges thought it would also raise awareness of the learning 
process and promote routes through lifelong learning. 

More than a few colleges recognised potential benefits of a national measure 
to them as institutions, including: 

• single point of reference for educational/societal development; 

• comparability for benchmarking; 

• assist in improving programme design; 

• common baseline, opportunity to publish results; 

• measures how well college makes a difference; and 

• useful as indicators, rather than measures. 
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Colleges were equally split on the benefits to employers and society of a 
distanced travelled measure. Those in favour saw benefits, through the 
provision of some measure of employability and soft skills. One college saw 
such national measures being linked to the principles within a Curriculum for 
Excellence, and learners developing the capacities to be: 

• successful learners; 

• confident individuals; 

• responsible citizens; and 

• effective contributors to society and the economy. 

However, another group of colleges were of the view that employers would 
find the measures confusing, and that employers are more interested in 
attainment outcomes than distance travelled. 

Overall, colleges considered that developing national measures would be 
challenging, difficult and highly complex, as well as expensive in terms of time 
and resources. The range of issues and barriers identified was very wide, 
with almost every response including caveats to the development. These 
included the following categories of comment that overall outweigh the more 
positive views of what could be achieved: 

• to implement the development of national measures would require the 
whole sector to sign up to the development, and there is unlikely to be 
general agreement on methods, definitions, etc.; 

• difficulties around developing a system to make national comparisons 
because of the subjectivity inherent in the measures; 

• difficult to identify the measure(s), quantify them and produce a reliable 
analysis of the outcomes. There would be serious barriers to achieving 
a uniform and useful measure and agreement would be unlikely; 

• issues relating to the very individual nature of the process of assessing 
distance travelled, including difficulty of establishing start points, 
difficulties in accurate self-assessment, the diversity of learner 
characteristics, and diversity of context; 

• costs in time and resources, nationally and to individual colleges, could 
be excessive and/or divert energies from other important areas; 

• the approach would need to incorporate a balance between flexibility 
(to best match needs of diversity of learners) and the uniformity and 
consistency required for national use. Fears that flexibility would be 
lost; 

• flaws in the existing Pis could be overstated if additional measures 
were in place; 

• concerns over the compulsory or voluntary nature of such a scheme; 
and 

• the need for staff development, and staff buy-in to the development. 

Several colleges made suggestions for ways of approaching some aspects of 
the issue. One suggested looking at the old PSD units. Another 
recommended the use of formal frameworks such as ASDAN, and SCQF, 
only where appropriate. Several colleges referred to SCQF, and while finding 
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it useful, a few found it limited in this context. One college made reference to 
the imperatives of a Curriculum for Excellence and the SFC’s Learning for All 
strategy. One comment, however, goes some way to summarising the 
general tenor of the responses: 

Requiring all colleges to have evidence of the discussion and recording 
of progress in terms of value added and distance travelled ‘soft skills’ 
might improve the practice of colleges not currently doing this. 

However, developing a formal mechanism for this would be fraught 
with difficulties in responding to different student groups and 
circumstances. 
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7. The use of benchmarking and trend analysis 


Approaches to benchmarking 

Colleges generally adopted three broad approaches to benchmarking 
attainment and achievement data: internal benchmarking, benchmarking in 
partnership with other colleges and external benchmarking against national 
data and other information. 

Overall, a range of internal benchmarking took place, although it was 
evident that benchmarking activity was far more embedded in some colleges 
than in others. Some colleges specified that benchmarking took place at 
programme or team level as well as in respect of global data. The range of 
activities included: 

• year-on-year internal comparisons and trends; 

• cross-college PI analysis; 

• comparisons with college targets; 

• inter-faculty comparisons; and 

• comparisons against national averages. 

About half the colleges were involved in benchmarking in partnership with 
other coiieges. The college groupings mentioned, with some examples of 
their activities were: 

• Aberdeen, Stow and Dumfries and Galloway Colleges - meetings three 
times a year to compare data and processes; 

• the Scottish Rural Colleges Network Academic Benchmarking Project; 

• the Quality Quadrant - Dundee, Glasgow Metropolitan, Jewel and Esk 
Colleges and Glasgow College of Nautical Studies - events for FE- 
level subject areas focusing on retention and achievement; 

• the six Lothian colleges; 

• the East-West colleges benchmarking club; 

• the Quality Managers’ Enhancement Forum (seven colleges) 
compared programmes with low Pis in 05-06 and set up subject 
networks to address issues; and 

• a benchmarking club comprising Cumbernauld, John Wheatley, North 
Glasgow and Langside Colleges. 

About half the colleges used external benchmarking against national data. 

Most of these cited SFC PI data and, in some cases, FES data supplied with 
the SFC benchmarking tool, and a few also used data from SQA. Several 
benchmarked against nationally published data for other colleges that 
resembled them in some way, for example the proportion of learners from the 
20% poorest SIMD zones or of similar size. A few also used HMIE college 
review and aspect reports, including the SLIPs, as a way of benchmarking 
against good practice. In addition, several colleges visited other colleges to 
compare practices, and one or two brought in Associate Assessors from other 
colleges to support informal quality improvement reviews. 
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Use of the SFC benchmarking tool 

Most colleges found the SFC benchmarking tool helpful, and a few had given 
all staff access to the tool. However, the tool had limitations at this stage of its 
development. Some colleges found it slow and cumbersome to use, not 
user-friendly, especially for staff unaccustomed to using spreadsheets. 
However, a few colleges indicated that problems had been resolved after 
consultation with or feedback to SFC, and many were optimistic about the tool 
and intended to use it more in future. Development visits by SFC staff to 
colleges to explain its use had been particularly helpful. 


Benchmarking focusing on iearner progress and outcomes: impact on 
quaiity 

Almost all colleges used their benchmarking activities to inform quality 
improvements. Retention, attainment and related measures were 
benchmarked, although few colleges benchmarked the achievement of soft 
skills. A majority of colleges used benchmarking to identify areas where there 
were weaknesses, in order to make improvements at programme team, 
department or college level. The concept of benchmarking was also widened 
to looking at good practice within or across colleges and used as a 
benchmark of standards to be reached. A few colleges also highlighted areas 
where there were strengths, to identify good practices and share them more 
widely. These colleges did this as part of programme/section review, or as a 
result of SLIPS published by HMIE. A few colleges also used benchmarking 
activity to feed into their planning processes, including programme design and 
team operational plans. 


Trend analysis 

Almost all colleges had embedded some form of trend analysis covering 
aspects of attainment, progress and achievement into their systems and 
processes, and this analysis had the capacity to inform teams at all levels. A 
majority of colleges carried out trend analysis at both programme and college 
level, and a few also included cognate groups or school/section level. 

A majority worked to a three-year cycle, but there were variations, partly 
according to the purposes for which the trend analysis was intended. For 
example, one college looked at three-year data at programme level and 
five-year data at college level. Another college looked at year on year data at 
all levels, but at three-year data for academic planning. And yet another 
college worked on a four-year cycle at programme and team level, looked at 
three years of data for the annual portfolio review process and had a full five- 
year curriculum review. 
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Most colleges used retention and SPAR data for trend analysis. A few 
colleges used student outcome. A few colleges also carried out trend 
analysis using PCSR or progression data and learner satisfaction survey data. 

In most colleges MIS or student records staff collected data, with analysis 
undertaken by academic staff or teams, with the involvement of quality 
management staff. 

In almost all colleges, trend analysis was embedded into the processes of 
self-evaluation and review, and linked to quality improvement. Colleges used 
trend analysis to: 

• inform programme design, focus on areas out of the normal trend, 
leading to programme review or change of portfolio; 

• shape the portfolio, identify delivery issues and review delivery 
methods; 

• diagnose problems; 

• focus on poor performance or good practice - trigger action planning; 

• monitor the effectiveness of interventions; and 

• set improvement targets in programme action plans and the college 
strategic plan. 
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8. Recommendations 


Colleges should: 

• use the early retention and student retention Pis as the measures of 
learner retention at programme and college level; 

• use the student outcome PI as the measure of programme attainment 
at programme and college level; 

• continue to use the SARU PI as the measure of unit attainment at 
programme and college level; 

• give close attention to a Curriculum for Excellence developments in 
recognising and accrediting learning across the four capacities, not just 
for 16-18 year old learners but for wider application; 

• build on current strengths and continue to develop tools and 
approaches to assess and record distance travelled that are credible 
and rigorous and can inform national developments; and 

• promote sharing of the outcomes of further development of tools and 
approaches to record and assess distance travelled and the use of 
benchmarking. 


To support developments nationally, SFC should: 

• engage with colleges and with SQA to address issues of consistent 
methodology for recording data; 

• engage with the colleges to establish the scope for developing helpful 
but non-burdensome indicators of distance travelled; and 

• ensure that future iterations of the quality framework used for external 
review continue to acknowledge both wider achievement and 
attainment to support holistic evaluations of learner progress and 
outcomes. 


HMIE should: 

• use the early retention, student retention, student outcome and SARU 
as the attainment PI measures in external reviews and other evaluative 
activities; and 

• identify, promote and disseminate further examples of good practice in 
assessing and recording learner progress and achievement across the 
wider college sector. 


Appendix 1. 


Performance indicator definitions 


Early student retention / SRR1 

Number of enrolments meeting funding gualifyinq date x 100 
Number of initial enrolments 


Student retention 

Number of these enrolments completing the 

programmes with student outcomes codes 7,8,9,14 and 15 x 100 

Number of enrolments meeting the funding qualifying date 


SRR2 

Number of these enrolments completing the 

programmes with student outcomes codes 7,8,9,14 and 15 x 100 

Number of initial enrolments 


SPAR 

Number of enrolments gaining the award x 100 
Number of enrolments meeting the funding 
qualifying date 


Student outcome 


Number of enrolments successful or progressing to next year x 100 
Number of enrolments completing the programme with 
student outcome codes 7,8,9,14 and 15 


SARU 

Number of assessment credits achieved x 100 
Number of assessment credits undertaken 


PCSR 


Number of successful students who gain employment 
or progress to more advanced education or training 
Number of successful students responding 


X 100 
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Appendix 2. Examples of good practice 


Adam Smith College: The use of PI data to inform decision making 

The college viewed PI data as a valuable starting point for discussions and 
planning for quality of learning and teaching, curriculum design, and overall 
improvement of the learner experience. Data was available to users on a 
real-time basis via the college’ electronic reporting system and incorporated a 
suite of reports covering retention and programme and unit attainment for 
different categories of learners. Data accuracy was monitored at point of 
entry by MIS staff for completeness and accuracy of keying. Programme 
teams used PI data effectively to underpin self-evaluation and action planning. 
Reliable and accurate data was central to all these discussions and was well 
understood by those involved in decision-making based upon it. The SFC 
benchmarking tool was also used well to benchmark against a range of 
comparator colleges at both subject and college level. Trend analysis of 
attainment data had been a focus on the design and delivery of the 
non-advanced curriculum, with particular focus on the provision of literacies to 
support progression and attainment through improved retention. 

Borders College: The use of SFC benchmarking tool in care 
programmes 

The staff team in care used the SFC benchmarking tool to support their 
quality enhancement programme. They identified those colleges in Scotland 
with high levels of retention and attainment in specific care programmes. The 
team arranged good practice visits to eight colleges to discuss with staff the 
approaches and strategies used to achieve successful learner progress and 
outcomes. The visits had been highly successful. The two-way exchange of 
practice had supported effective professional dialogue. Following the visits, 
the team had adjusted aspects of the design of the programmes and 
enhanced levels of learner support, particularly in the early stages of 
programmes and where learners were at risk of dropping out. The visits had 
helped staff address the findings of the HMIE external review of care in 2006. 
Staff had made good contacts at the colleges concerned and a strong basis 
for future discussion and sharing of practice. 

Jewel and Esk College: Post course success analysis 

The college gave priority to the collection and reporting of learner 
destinations. It routinely achieved a response rate of approximately 90%. 
Successful strategies to increase response rates included contacting learners 
at the college’s annual graduation ceremony using a call centre for a 
two-week period including evenings. The college had developed a 
progression viewer staff that provided easy access to employment, 
progression and articulation information. The progression v/ewer provided a 
user-friendly front end to the college’s destinations database. It allowed the 
user to access possible university articulation routes for a specific programme 
as well as the actual employment and educational routes for past successful 
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college learners. Employment details included type of employment and 
employer, whether the employment was full-time or part-time, related or 
unrelated to the programme studied, and the salary range. The educational 
route section displayed the college or university learners had progressed to 
and the programme they had enrolled on. Staff used the progression viewer 
successfully to inform programme marketing, planning and evaluation. 

Jewel and Esk College: Online approaches to core skills 

The college had developed an online approach for delivering core skills to 
learners on HN programmes and accrediting both their achievement and the 
distance travelled. It had applied the approach to four HNC frameworks: 
social care, multi media, electrical engineering, and working with 
communities. Staff had found that these programmes often had no core skills 
units in the frameworks and that learners’ skills were not at the level required 
to successfully complete other units. The college had worked collaboratively 
with SQA to develop the HN Personal Development Planning unit using the 
college VLE and the SQA Academy website. The work aimed to facilitate 
programme design and delivery and provide teaching and learning to deliver 
fully the researched core skills exit profile within HN frameworks and 
programmes. Staff had found that the use of the college’s VLE had enhanced 
learner participation and motivation and provided an attractive means of 
assessment through e-portfolio and accrediting achievement. 

John Wheatley College: The use of ILPs to record and assess learner 
progress 

The college had developed a useful independent learning planning process 
which provided a solid baseline on which to assess the progress made by 
learners during their studies. An Individual Learning Plan (ILP) was built up 
by learners during the pre-entry and induction periods, recording prior 
attainment and personal learning goals. All learners undertaking full-time and 
substantial part-time programmes undertook core skills screening and 
vocational skills assessment. These outcomes were recorded on the ILP. 
Progress against personal learning goals, attainment of certificated units and 
achievement of citizenship and employability skills were routinely reviewed 
through the academic year. This enabled learners and staff to revise the 
learning goals taking account of progress and to identify support needs as 
they emerged. The college ran a range of quality enhancement activities, 
including learner focus groups, to assess the progress made by learners in 
achieving both citizenship and employability skills. These skills were in line 
with the four capacities within a Curriculum for Excellence. Pre-exit guidance 
activities reviewed overall learner achievement to identify progression routes 
for each individual learner. Learners benefitted from this holistic approach to 
assessing attainment and wider achievement. 
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Appendix 3. The use of performance indicators on iearner 
progress in post-16 provision in Engiand and 
Waies 


The UK government has set out the plans for the development of a single 
framework, Framework for Excellence, which provides benchmarked and 
validated assessment of performance in English colleges. A pilot was carried 
out in 2007/08 and from 2008/09 the Framework will apply to all colleges and 
work-based learning providers. There is a core set of Pis that is the minimum 
to provide a comprehensive picture of performance to all stakeholders. 

An overall performance rating is calculated for each college using: 

• performance in a range of areas through the use of Pis organised into 
seven key performance areas; and 

• college or providers grades which are aggregated to produce grades 
for three dimensions - responsiveness, effectiveness and finance. 

The effectiveness dimension is informed by data from two key performance 
areas: 

• quality of outcomes; and 

• quality of provision. 

The grade for the performance indicator, quality of outcomes, is derived from 
four qualification success rates (QSRs). These are success rates for long 
courses in FE; short courses in FE; A level courses; and apprenticeship and 
advanced apprenticeship scores. The A level QSR is supplemented by the 
value-added outcome from the Learning and Skills Council’s (LSCs) Learner 
Achievement Tracker (LAT). This shows how much progress in qualifications 
individual 16-19 year old learners have made, based on their prior attainment, 
compared to national results. 

The scores for the four constituent elements in the quality of outcomes key 
performance area is converted into one score using a scoring system. A 
weighted average of the scores for the four areas in quality of outcomes and 
the grade for quality of provision is calculated to give a grade of between 1 
and 4. In the quality of provision key performance area the key PI will be 
Ofsted’s current judgement on the overall effectiveness of the college or 
provider. The grades are shown in the table on the next page: 
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Standard for the dimension 

Assessment criteria 

Grade 1 

Outstanding 

Sum of grades for constituent key 
performance areas is 2 

Grade 2 

Good 

Sum of grades for constituent key 
performance areas is 3 or 4 

Grade 3 

Satisfactory 

Sum of grades for constituent key 
performance areas is 5 or 6 

Grade 4 

Inadequate 

Sum of grades for constituent key 
performance areas is 7 or 8 


Ofsted is currently working with partner organisations to ensure a coherent 
relationship between the Framework for Excellence and the GIF within the 
target date of 2009. 

Under the current Ofsted reporting system, using the Common Inspection 
Framework (GIF), all colleges will be inspected at least once between 2005 
and 2009. At present colleges normally receive three working weeks notice of 
their inspection. The timing and level of an inspection depends mainly on: 

• the date of the college’s last inspection or re-inspection; 

• the quality of provision and standards of performance, as reflected in 
the grades achieved at the last inspection, and the college’s 
performance since that time; and 

• other information resulting from monitoring visits or desk monitoring 
advice from the local or national learning and skills council. 

The GIF has been revised to cover five key questions. 

1 . How well do learners achieve? 

2. How effective are teaching, training and learning? 

3. How well do programmes and activities meet the needs and interests 
of the learners? 

4. How well are learners guided and supported? 

5. How effective are leadership and management in raising achievement 
and supporting all learners? 

Inspectors look at data for learners who have completed their programmes in 
the last three years. They give more weight to the most recent years and to 
courses that are currently operating and consider retention rates for existing 
learners and results of any mid-course external tests, examinations and 
course work. 

One hundred providers were selected for the pilot between September 2007 
and March 2008. The pilot tested the assessment criteria and the 
aggregation from Pis to key performance area, dimension and overall rating. 
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Appendix 5. Colleges participating in this study 


The following colleges provided information and views to inform this report: 

Aberdeen College 
Adam Smith College 
Angus College 
Anniesland College 
Ayr College 

Banff and Buchan College 
Borders College 
Cardonald College 

Carnegie College (formerly Lauder College) 

Central College 
Coatbridge College 
Cumbernauld College 
Dumfries and Galloway College 
Edinburgh’s Telford College 
Elmwood College 

Glasgow College of Nautical Studies 
Glasgow Metropolitan College 
Inverness College 

James Watt College of Further and Higher Education 

Jewel and Esk College 

John Wheatley College 

Kilmarnock College 

Langside College 

Motherwell College 

Newbattle Abbey College 

North Glasgow College 

North Highland College 

Oatridge College 

Perth College 

Reid Kerr College 

Sabhal Mor Ostaig 

South Lanarkshire College 

Stevenson College Edinburgh 

Stow College 

West Lothian College 


35 


